Image processing apparatus, image processing method, image processing program, recording medium recording the image processing program, and moving object detection system

ABSTRACT

It is intended to provide an image processing apparatus enabling an image processing for association of a standard image to a reference image using a reduced number of pixels without deteriorating accuracy for detecting a movement of a detection subject. A detection subject region setting unit obtains a picked-up image at a first time point as a standard image from time-series picked-up image data obtained by an image pickup unit and selects a plurality of correlation windows including an image region of the detection subject in the standard image with relative positions with respect to the standard point being identified. A movement detection unit obtains a picked-up image at a second time point that is different from the first time point as a reference image and detects a movement of the detection subject between the standard image and the reference image by calculating in the reference image a position of the standard point that minimizes a difference between a state of pixel values of the correlation windows of the standard image and a state of pixel values of correlation windows of the reference image.

This application claims priority from Japanese patent application122570/2006, filed on Apr. 26, 2006. The entire contents of theaforementioned application is incorporated herein by reference.

TECHNICAL FIELD

This invention relates to an image processing apparatus for detectingmovement of a detection subject such as a person or an object, an imageprocessing method, an image processing program, and a recording mediumrecording the image processing program.

BACKGROUND ART

Moving object tracking systems that pick up a movement of a trackingsubject such as a person or an object as a moving image using asurveillance camera and determine a movement of the picked up person orobject by analyzing the moving image have heretofore been provided. Themoving object tracking system is usable for traffic control, vehicletravel assistance, and the like by determining movements of vehicles,for example. Also, the moving object tracking system is usable forpassersby status investigation, customer behavior investigation, and thelike by determining movements of passersby in public areas and movementsof customers and the like in retail premises. Also, the moving objecttracking system is usable for production control, production planning,and the like by determining movements of production subjects in aproduction line in a plant and movements of workers.

Such moving object tracking system is provided with an image processingapparatus for performing an image processing on picked-up images whichhave been picked up by the surveillance camera. Such image processingapparatus detects movements of a detection subject by performing aprocessing of associating the detection subject in a current image(reference image) to the detection subject in a past image (referenceimage) as an identical object.

In the association processing, a region including an entire image regionof the detection subject is set as a correlation window based on thereference image. By performing a search in the reference image with theuse of the correlation window, a position of the correlation window atwhich a difference between pixel values in the correlation window in thestandard image and pixel values in the correlation window in thereference image is minimized is determined as a position of thedetection subject in the reference image.

-   [Patent Publication] JP-A-2003-346159 (published on Dec. 5, 2003).

SUMMARY OF THE INVENTION

In the case of performing the detection subject movement detectionprocessing on real-time bases, it is necessary to reduce a processingtime. In the case where a speed of the movement of the detection subjectis high, it is necessary to increase the number of frames per unit timein a moving image, and spacing between frames is shortened to make itnecessary to complete the association in the short inter-frame spacing.On the other hand, when the inter-frame spacing for performing themovement detection processing is increased for the purpose of ensuringthe processing time, a movement distance of the detection subject duringthe spacing is increased, thereby increasing a risk of failing to detecta movement depending on a speed of the detection subject. Accordingly,the time required for the movement detection processing is furtherincreased.

As described above, the movement detection processing is performed byusing information of values of pixels included in the correlationwindow. Therefore, the processing time is increased with an increase innumber of pixels included in the correlation window.

In general, the correlation window is so set as to include an entirepart of the detection subject (for example, an entire part of a person).In such case, the number of pixels included in the correlation window isincreased to result in an increase in processing time.

Patent Publication 1 discloses a technology for stably detecting amovement of a person by setting a part of the person such as a head anda trunk of the person as a tracking subject. In this case, though theprocessing time is reduced as compared to the case of setting an entirepart of the person as the tracking subject, the number of pixels isstill large, and the technology is not so practical from the stand pointof the real-time processing.

This invention was accomplished in view of the above-described problems,and an object thereof is to provide an image processing apparatus, animage processing method, an image processing program, and a recordingmedium recording the image processing program, which enable to performan image processing on a reduced number of pixels for associating astandard image to a reference image without deteriorating accuracy fordetecting a movement of a detection subject.

In order to solve the above-described problems, an image processingapparatus according to this invention has a structure of comprising apartial image region setting unit for obtaining a picked-up image at afirst time point as a standard image from time-series picked-up imagedata obtained by an image pickup unit picking up a detection subject andselecting a plurality of partial image regions including an image regionof the detection subject in the standard image and a movement detectionunit for obtaining a picked-up image at a second time point that isdifferent from the first time point as a reference image and detecting amovement of the detection subject between the standard image and thereference image based on a difference between a state of pixel values ofthe partial image regions of the standard image and a state of pixelvalues of partial image regions of the reference image, wherein thepartial image region setting unit sets a standard point used as astandard for relative positions of the partial image regions; and themovement detection unit obtains the state of the pixel values of thepartial image regions of the reference image with the relative positionsof the partial image regions with respect to the standard point beingunchanged and calculates the difference between the obtained state ofthe pixel values of the partial image regions and the state of the pixelvalues of the partial image regions of the standard image.

Also, in order to solve the above-described problems, an imageprocessing method according to this invention comprises a partial imageregion setting step for obtaining a picked-up image at a first timepoint as a standard image from time-series picked-up image data obtainedby an image pickup unit picking up a detection subject and selecting aplurality of partial image regions including an image region of thedetection subject in the standard image and a movement detection stepfor obtaining a picked-up image at a second time point that is differentfrom the first time point as a reference image and detecting a movementof the detection subject between the standard image and the referenceimage based on a difference between a state of pixel values of thepartial image regions of the standard image and a state of pixel valuesof partial image regions of the reference image, wherein a standardpoint used as a standard for relative positions of the partial imageregions is set in the partial image region setting step; and the stateof the pixel values of the partial image regions of the reference imageare obtained with the relative positions of the partial image regionswith respect to the standard point being unchanged to calculate thedifference between the obtained state of the pixel values of the partialimage regions and the state of the pixel values of the partial imageregions of the standard image in the movement detection step.

According to the above-described structure and method, the partial imageregions including the image region of the detection subject in thestandard image are selected at plural positions. After that, the stateof the pixel values in the plural partial image regions of the standardimage is compared with that of the reference image, and a movement ofthe detection subject between the standard image and the reference imageis detected based on the thus-detected difference.

Since the partial image regions are set at the plural positions on thedetection subject, it is possible to accurately determinecharacteristics of a shape of the detection subject when the numbers ofpixels included in the partial image regions are reduced. Accordingly,as compared to the method of detecting a movement of a detection subjectby associating an image region including an entire image region of thedetection subject of a standard image to that of a reference image, itis possible to perform the image processing on a reduced number ofpixels for associating the standard image to the reference image.Therefore, it is possible to provide the image processing apparatusrealizing the detection of the movement of the detection subject in areduced processing time.

Also, according to the above-described structure, the state of the pixelvalues in the partial image regions of the reference image is obtainedwith the relative positions of the partial image regions with respect tothe standard point being unchanged. That is, it is possible to associatethe detection subject in the standard image to that in the referenceimage with the positional relationship between the plural positionscharacterizing the shape of the detection subject being considered.Therefore, it is possible to accurately perform the movement detectionof the detection subject in the case where the shapes of the detectionsubjects in the standard image and the reference image differ littlefrom each other.

In order to solve the above-described problems, an image processingapparatus according to this invention has a structure of comprising apartial image region setting unit for obtaining a picked-up image at afirst time point as a standard image from time-series picked-up imagedata obtained by an image pickup unit picking up a detection subject andselecting a plurality of partial image regions including an image regionof the detection subject in the standard image and a movement detectionunit for obtaining a picked-up image at a second time point that isdifferent from the first time point as a reference image and detecting amovement of the detection subject between the standard image and thereference image based on a difference between a state of pixel values ofthe partial image regions of the standard image and a state of pixelvalues of partial image regions of the reference image, wherein thepartial image region setting unit sets a standard point used as astandard for relative positions of the partial image regions; and themovement detection unit obtains the state of the pixel values of thepartial image regions of the reference image by converting the relativepositions of the partial image regions with respect to the standardpoint by a predetermined conversion processing and calculates a positionof the standard point in the reference image by calculating thedifference between the obtained state of the pixel values of the partialimage regions and the state of the pixel values of the partial imageregions of the standard image.

Also, in order to solve the above-described problems, an imageprocessing method according to this invention comprises a partial imageregion setting step for obtaining a picked-up image at a first timepoint as a standard image from time-series picked-up image data obtainedby an image pickup unit picking up a detection subject and selecting aplurality of partial image regions including an image region of thedetection subject in the standard image and a movement detection stepfor obtaining a picked-up image at a second time point that is differentfrom the first time point as a reference image and detecting a movementof the detection subject between the standard image and the referenceimage based on a difference between a state of pixel values of thepartial image regions of the standard image and a state of pixel valuesof partial image regions of the reference image, wherein a standardpoint used as a standard for relative positions of the partial imageregions is set in the partial image region setting step; and the stateof the pixel values of the partial image regions of the reference imageby converting the relative positions of the partial image regions withrespect to the standard point by a predetermined conversion processingis obtained to calculate a position of the standard point in thereference image by calculating the difference between the obtained stateof the pixel values of the partial image regions and the state of thepixel values of the partial image regions of the standard image in themovement detection step.

According to the above-described structure and method, the partial imageregions including the image region of the detection subject in thestandard image are selected at plural positions. After that, the stateof the pixel values in the plural partial image regions of the standardimage is compared with that of the reference image, and a movement ofthe detection subject between the standard image and the reference imageis detected based on the thus-detected difference.

Since the partial image regions are set at the plural positions for thedetection subject, it is possible to accurately determinecharacteristics of a shape of the detection subject when the numbers ofpixels included in the partial image regions are reduced. Accordingly,as compared to the method of detecting a movement of a detection subjectby associating an image region including an entire image region of thedetection subject of a standard image to that of a reference image, itis possible to perform the image processing on the reduced number ofpixels for associating the standard image to the reference image withoutdeteriorating an accuracy for detecting the movement of the detectionsubject. Therefore, it is possible to provide the image processingapparatus realizing the detection of the movement of the detectionsubject in a reduced processing time.

Also, according to the above structure or the method, the relativepositions of the partial image regions with respect to the standardpoint are converted by the predetermined conversion processing whenobtaining the state of the pixel values in the partial image regions ofthe reference image. Therefore, it is possible to accurately associatethe detection subjects of the standard image and the reference image toeach other in the case where a size or an orientation of the image ofthe detection subject of the standard image is different from that ofthe reference image.

Also, in the above-described structure, the image processing apparatusaccording to this invention may have a structure that a total of numbersof pixels in the plural partial image regions is smaller than the numberof pixels of the image region of the detection subject. With suchstructure, it is possible to more reliably realize the reduction inprocessing time.

Also, in the above-described structure, the image processing apparatusaccording to this invention may have a structure that the partial imageregion setting unit divides the image region of the detection subjectinto a plurality of intermediate regions to select at least one of theintermediate regions as the partial image region. With such structure,it is possible to perform the association based on the entire shape ofthe detection subject by selecting the partial image regions evenly fromthe intermediate regions.

Also, in the above-described structure, the image processing apparatusaccording to this invention may have a structure that the partial imageregion setting unit selects a plurality of characteristic points byapplying a filter for detecting the characteristic points characterizinga partial shape of the detection subject to the standard image to set aregion including the characteristic points as the partial image region.

According to the above-described structure, the characteristic pointsare selected by applying the filter to the standard image. Since thesize required for detecting the characteristic points characterizingpartial shapes of the detection subject is sufficient as the size ofthis filer, the filter has a relatively small size. Therefore, theprocessing time required for detecting the characteristic points becomesrelatively short. Specifically, examples of the filter for detecting thecharacteristic points characterizing the partial shape of the detectionsubject are a primitive filter such as those used for detecting ahorizontal edge and an angle edge. Since the size of such filters isrelatively small, it is possible to rapidly perform the characteristicpoint detection.

Also, in the above-described structure, the image processing apparatusaccording to this invention may have a structure that the movementdetection unit performs a processing of enlarging or reducing a distancebetween each of the relative positions of the partial image regions andthe standard point as the predetermined conversion processing.

According to the above-described structure, in the case where the sizeof the image of the detection subject in the standard image differs fromthat of the reference image, it is possible to accurately perform theassociation in response to the change.

Also, in the above-described structure, the image processing apparatusaccording to this invention may have a structure that the movementdetection unit decides an enlargement/reduction ratio based on aposition of the detection subject in the standard image and a positionof the detection subject in the reference image.

For example, in the case where the image pickup unit is disposed fixedlyand the detection subject moves in a two-dimensional plane, it ispossible to determine a distance between the detection subject and theimage pickup unit by the position of the detection subject in thepicked-up image. With the use of the distance determination of theabove-described structure, it is possible to largely reduce the amountoperation for calculating the position of the partial image region inwhich the difference between the state of the pixel values in thepartial image regions of the standard image and the state of the pixelvalues in the partial image regions of the reference image is minimizedby deciding the enlargement/reduction ratio based on the position of thedetection subject in the standard image and the position of thedetection subject in the reference image. That is, the above structureachieves its effect when the image pickup unit is disposed fixedly andthe detection subject moves in the two-dimensional plane, particularlywhen the image pickup unit is a surveillance camera mounted on aceiling.

Also, in the above-described structure, the image processing apparatusaccording to this invention may have a structure that the movementdetection unit obtains from the image pickup unit two picked-up imagesobtained by picking up the detection subject from two different pointssimultaneously and calculates a distance between the detection subjectand the image pickup unit in the standard image and a distance betweenthe detection subject and the image pickup unit in the reference imagebased on the two picked-up images to decide the enlargement/reductionratio based on the distances.

According to the above-described structure, the distance between thedetection subject and the image pickup unit in the standard image andthe distance between the detection subject and the image pickup unit inthe reference image are calculated based on the two picked-up imagessimultaneously obtained by picking up the detection subject from thepositions different from each other. By deciding theenlargement/reduction ratio based on the distances, it is possible tolargely reduce the amount of operation for calculating the position ofthe partial image region in which the difference between the state ofthe pixel values in the partial image regions of the standard image andthe state of the pixel values in the partial image regions of thereference image is minimized by deciding the enlargement/reduction ratiobased on the distances. That is, the above-described structure achievesits effect when the image pickup unit and the movable place of thedetection subject are not restricted, particularly when the image pickupunit is an on-vehicle stereo camera for monitoring a frontwarddirection.

Also, in the above-described structure, the image processing apparatusaccording to this invention may have a structure that the movementdetection unit performs as the predetermined conversion processing aprocessing of rotating the relative positions of the partial imageregions, which are relative to the standard point.

According to the above-described structure, when the orientation of thedetection subject in the standard image is different from that of thereference image, it is possible to accurately associate the detectionsubjects to each other in response to the difference.

Also, in the above-described structure, the image processing apparatusaccording to this invention may have a structure that the movementdetection unit decides a rotation angle based on the position of thedetection subject in the standard image and the position of thedetection subject in the reference image.

For example, when conditions for the orientation of the detectionsubject are identified by the position of the detection subject in thepicked-up image, it is possible to uniquely decide the rotation angle bypreliminary setting a rotation relationship between the regions in thepicked-up image. With the use of such rotation relationship, by decidingthe position of the detection subject in the standard image and theposition of the detection subject in the reference image as describedabove, it is possible to largely reduce the amount operation forcalculating the position of the partial image region at which thedifference between the state of the pixel values in the partial imageregions of the standard image and the state of the pixel values in thepartial image regions of the reference image is minimized.

Also, in the above-described structure, the image processing apparatusaccording to this invention may have a structure that the movementdetection unit obtains the state of the pixel values of the partialimage regions by rotating the partial image regions of the referenceimage.

When an orientation of the detection subject is changed, orientations ofthe images in the partial image regions also change. In order to dealwith such orientation change, not only relative coordinates of thepartial image regions but also the images in the partial image regionsare rotated. Therefore, it is possible to more accurately associate thedetection subject in the standard image to that of the reference imagein the case where the orientation of the detection subject changeslargely.

The above-described image processing unit may be realized by using acomputer, and, when the image processing apparatus is realized with theuse of the computer, this invention encompasses an image processingprogram that causes the computer to realize the image processingapparatus by causing the computer to operate as the above-describedunits and a recording medium recording the image processing program andreadable by the computer.

As described above, the image processing apparatus according to thisinvention has a structure of comprising a partial image region settingunit for obtaining a picked-up image at a first time point as a standardimage from time-series picked-up image data obtained by an image pickupunit picking up a detection subject and selecting a plurality of partialimage regions including an image region of the detection subject in thestandard image and a movement detection unit for obtaining a picked-upimage at a second time point that is different from the first time pointas a reference image and detecting a movement of the detection subjectbetween the standard image and the reference image based on a differencebetween a state of pixel values of the partial image regions of thestandard image and a state of pixel values of partial image regions ofthe reference image, wherein the partial image region setting unit setsa standard point used as a standard for relative positions of thepartial image regions; and the movement detection unit obtains the stateof the pixel values of the partial image regions of the reference imagewith the relative positions of the partial image regions with respect tothe standard point being unchanged and calculates the difference betweenthe obtained state of the pixel values of the partial image regions andthe state of the pixel values of the partial image regions of thestandard image to calculate a position of the standard point in thereference image.

As described above, the image processing apparatus according to thisinvention has a structure of comprising a partial image region settingunit for obtaining a picked-up image at a first time point as a standardimage from time-series picked-up image data obtained by an image pickupunit picking up a detection subject and selecting a plurality of partialimage regions including an image region of the detection subject in thestandard image and a movement detection unit for obtaining a picked-upimage at a second time point that is different from the first time pointas a reference image and detecting a movement of the detection subjectbetween the standard image and the reference image based on a differencebetween a state of pixel values of the partial image regions of thestandard image and a state of pixel values of partial image regions ofthe reference image, wherein the partial image region setting unit setsa standard point used as a standard for relative positions of thepartial image regions; and the movement detection unit obtains the stateof the pixel values of the partial image regions of the reference imageby converting the relative positions of the partial image regions withrespect to the standard point by a predetermined conversion processingand calculates a position of the standard point in the reference imageby calculating the difference between the obtained state of the pixelvalues of the partial image regions and the state of the pixel values ofthe partial image regions of the standard image.

As described above, since the image processing is performed on thereduced number pixels for associating the standard image to thereference image without deteriorating detection accuracy, an effect ofenabling to provide the image processing apparatus that realizes adetection of movement of a detection subject in a reduced processingtime is achieved.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a schematic structure of a movingobject tracking system according to one embodiment of this invention.

FIGS. 2( a) and 2(b) are diagrams showing states in which pluralcorrelation windows are applied to a standard image and a referenceimage.

FIG. 3 is a graph showing results obtained by calculating evaluatedvalues by moving the correlation windows in an X-axis direction byemploying methods of Comparative Example and this embodiment.

FIG. 4 is a flowchart showing a flow of a movement detection processing.

FIG. 5 is a diagram showing one example of image of the detectionsubject and one example of detection subject region set by a detectionsubject region setting unit.

FIGS. 6( b) to 6(e) are diagrams showing results obtained by applyingfour filters to the detection subject region, and FIG. 6( a) is adiagram showing a result obtained by selecting a minimum value from thefiltering result values in the processing results shown in FIGS. 6( b)to 6(e).

FIGS. 7( a) and 7(b) are diagrams showing states of central coordinatesof the plural correlation windows and the detection subject in thestandard image and the reference image.

FIG. 8 is a diagram showing a change in size of the picked-up image inthe case where the detection subject moves in such a manner as toapproach to the image pickup unit.

FIGS. 9( a) and 9(b) are diagrams showing a state of Comparative Examplein which one correlation window is set in each of the standard image andthe reference image in the case where the size of the detection subjectchanges.

FIGS. 10( a) and 10(b) are diagrams showing a state of this embodimentin which the plural correlation windows are set in each of the standardimage and the reference image in the case where the size of thedetection subject changes.

FIG. 11 is a side view showing an image pickup environment in which theimage pickup unit is disposed as being fixed with respect to thedetection subject moving on a plane.

FIG. 12 is a diagram showing an image pickup environment in which twoimage pickup units are provided.

FIG. 13( a) is a diagram showing, in the case where the detectionsubject is a car, a state in which the car is picked up when the car isapproaching to the image pickup unit while rounding a curve, and FIG.13( b) is a diagram showing a binary image of the image of FIG. 13( a).

FIG. 14( a) is a diagram showing, in the case where the detectionsubject is a car, one example in which the car is traveling on a roadhaving a branch point, and FIG. 14( b) is a diagram showing a region ofthe road included in the picked-up image of FIG. 14( a).

FIG. 15 is an enlarged view of one of the correlation windows in theprevious and current images of the detection subject shown in FIG. 13(b).

FIGS. 16( a) to (d) are diagrams showing a process of selectingcharacteristic points and setting correlation windows in the case wherea car is picked up as the detection subject.

FIG. 17 is a diagram showing filters applied to the image shown in FIG.16( a).

FIG. 18 is a diagram showing one example of filter to be used for thecase of detecting a movement of a person as a detection subject.

FIG. 19 is a diagram showing a process for selecting the characteristicpoints in the case of detecting a movement of a person as a detectionsubject.

FIG. 20 is a flowchart showing another example of flow of the movementdetection processing.

FIGS. 21( a) to 21(c) are diagrams showing examples of standard image,reference image, and movement tracking display image in the case whereplural detection subjects exist in each of the images.

FIG. 22( a) is a diagram showing one example of state of movement of thedetection subject in a moving image, and FIGS. 22( b) and 22(c) arediagrams showing a standard image and a reference image of ComparativeExample, to each of which one correlation window is applied.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, one embodiment of this invention will be described based onthe drawings.

(Structure of Moving Object Tracking System)

FIG. 1 is a block diagram showing a schematic structure of a movingobject detection system 1 according to this embodiment. This movingobject detection system 1 is provided with an image pickup unit 2, animage processing unit (image processing apparatus) 3, and a display unit4 as shown in FIG. 1.

The image pickup unit 2 picks up a detection subject which is a subjectof movement detection and the image pickup is performed with the use ofa CCD (charge coupled device) image pickup element, a CMOS(complementary metal oxide semiconductor) image pickup element, or thelike. The image processing unit 3 recognizes the detection subjectincluded in the picked-up image based on the picked-up image obtained bythe image pickup unit 2 to perform an image processing for detecting amovement of the detection subject. The display unit 4 displays thepicked-up image obtained by the image pickup unit 2 as well as amovement detection result of the detection subject obtained by the imageprocessing unit 3 and is provided with various display devices capableof displaying images such as a CRT (Cathode Ray Tube) and a liquidcrystal display device.

The image processing unit 3 is provided with a picked-up image storagecontrol unit 11, a detection subject region setting unit 12, acorrelation window setting unit 13, a movement detection unit 14, anoutput control unit 15, an image storage unit 21, a detection subjectregion storage unit 22, a correlation window storage unit 23, and amovement detection information storage unit 24.

The picked-up image storage control unit 11 receives moving image datafrom the image pickup unit 2 as time-series picked-up image dataobtained by the image pickup unit 2 and perform a processing for storingthe moving image data in the image storage unit 21. The detectionsubject region setting unit 12 takes out the picked-up image stored inthe image storage unit 21 to extract the detection subject and performsa processing for setting a detection subject region including an imageregion extracted as the detection subject. Information of the setdetection subject region is stored in the detection subject regionstorage unit 22.

The correlation window setting unit 13 selects plural characteristicpoints from the detection subject region set by the detection subjectregion setting unit 12 and performs a processing for setting acorrelation window (partial image region) for each of the characteristicpoints. Information of the set correlation windows is stored in thecorrelation window storage unit 23.

The movement detection unit 14 acquires the picked-up images stored inthe image storage unit 21 and performs an evaluation operation using thecorrelation windows set by the correlation window setting unit 13,thereby associating the detection subjects to each other in thepicked-up images as well as detecting a movement of the detectionsubject. Information of the detected detection subject movement isstored in the movement detection information storage unit 24.

The output control unit 15 performs control for causing the picked-upimages stored in the image storage unit 21 to be displayed by thedisplay unit 4 as well as control for causing the information indicatingthe movement of the detection subject detected by the movement detectionunit 14 to be displayed by the display unit 4. The display of thepicked-up images may be performed substantially simultaneously with theimage pick-up by the image pickup unit 2 on real-time basis or may beperformed at a time point after the image pickup.

Though the picked-up images and the movement detection information aredisplayed on the display unit 4 by the output control unit 15 in thisembodiment, the output control unit 15 may have any structure insofar asthe output control unit 15 outputs the picked-up images and the movementdetection information. For example, the output control unit 15 may havea structure that the picked-up images stored in the image storage unit21 and the movement detection information stored in the movementdetection information storage unit 24 are sent to an external device sothat the movement detection result is confirmed in the external device.

COMPARISON WITH COMPARATIVE EXAMPLE

A comparison between a case of setting a region including an entireimage region of a detection subject in one correlation window and a caseof setting plural correlation windows are set as in the embodiment ofthis invention will be described. As Comparative Example, a case ofsetting a region including an entire image region of a detection subjectin a correlation window RW will be described as Comparative Example withreference to FIGS. 22( b) and 22(c), and then the method of settingcorrelation windows in this embodiment will be described with referenceto FIGS. 2( a) and 2(b).

Shown in FIG. 22( a) is a state of movement of a detection subject in amoving image. In this example, a state in which the detection subjectmoves from a position A to a position B is shown.

Shown in FIG. 22( b) is a state (standard image) of pixel values whenthe detection subject is at the position A, and shown in FIG. 22( c) isa state (reference image) of pixel values when the detection subject hasmoved to the position B. In order to simplify the description, the pixelvalue is represented by binary values of 0 or 1. Also, a pixel value ofa region in which the detection subject exists is represented by 0, anda pixel value of a region other than the region in which the detectionsubject exists is represented by 1.

In Comparative Example, a region including the entire image region ofthe detection subject is set as the correlation window RW as shown inFIG. 22( b) based on the standard image. Then, a correlation window RWfor a central pixel C (xc, yc) is searched in the reference image. Morespecifically, an evaluation operation is performed by setting acorrelation window RW′ having the size identical to that of thecorrelation window RW for each of pixels (x′c, y′c) in the referenceimage with the pixel (x′c, y′c) being set as the center of thecorrelation window RW′.

For example, when assuming the case wherein the detection subject movesonly in a horizontal direction (X-axis direction) in the image data, thefollowing expression is set as one example of evaluation function.

[Expression 1]

In Expression 1, k represents a pixel value (x0, y0: original pint ofcorrelation window coordinate system) in coordinates in the correlationwindow RW of the standard image, and k′ represents a pixel value (x′0,y′0: original pint of correlation window coordinate system) incoordinates in the correlation window RW′ of the reference image. Also,M and N represents a horizontal pixel number and a vertical pixel numberof the set correlation window. When each of M and N is an odd number, avalue of M/2 or the like is appropriately converted into an integer.

In this case, the position of the central pixel when the evaluated valueE is minimum is determined as the center position of the detectionsubject region, and the position of RW′ when the evaluated value E isminimum is determined as the position of the detection subject in thereference image.

When moving the correlation window RW in the X-axis direction, theposition of the correlation window RW at which the evaluated value E(x′c, y′c) obtained by Expression 1 is minimized is determined as theposition of the detection subject in the reference image.

Though only the movement in the X-axis direction is considered in theabove example, it is possible to detect a movement in an arbitrarydirection in a two dimensional plane by considering a movement in Y-axisdirection.

In Comparative Example, the correlation window RW having the size ofincluding the entire image region of the detection subject is set on thestandard image. That is, the operation of Expression 1 is performed byusing the pixel values of the entire pixels included in the correlationwindow RW. As is apparent from Expression 1, a processing time isincreased with an increase in the number of pixels subjected to theoperation. Therefore, in order to shorten the processing time, it isdesirable to reduce the number of pixels for the operation.

Shown in FIG. 2( a) is a state of employing the method of setting thecorrelation window in this embodiment. As shown in FIG. 2( a), fourcorrelation windows RWA to RWD are set on four small regions eachincluding a part of the image of the detection subject.

After that, a pixel C (xc, yc) which is the center of the correlationwindows RWA to RWD is set as a standard point, and a corresponding pointfor the standard point is searched in a reference image. Morespecifically, an evaluation operation is performed by settingcorrelation windows RWA′ to RWD′ having the size identical to those ofthe correlation windows RWA to RWD around pixels (xc′, yc′) in thereference image at relative positions which are identical to those ofthe correlation windows RWA to RWD. Specifically, by calculating in thereference image a position of a standard point c′ defining thecorrelation windows RWA′ to RWD′ that minimize a difference with thepixel values in the correlation windows RWA to RWD, and a position ofthe detection subject in the reference image is calculated based on theposition of the standard point c′.

As described above, the movement detection is performed by using theplural small correlation windows each including a part of the detectionsubject in this embodiment. That is, it is possible to preciselydetermine characteristics of the detection subject by using the pluralcorrelation windows, and, it is possible to reduce the number of pixelsin each of the correlation windows to that required for determining apart of the characteristics of the detection subject. Accordingly,though the number of correlation windows is increased, it is possible tolargely reduce the number of pixels included in each of the correlationwindows, and, as a result, it is possible to reduce the number of pixelssubjected to the operation. Therefore, it is possible to realize areduction in processing time.

In this invention, since the standard point is different from thecentral point of each of the correlation windows, it is impossible tocalculate the evaluated value E by using Expression 1 as it is.Accordingly, the inventors of this invention performed expansion asindicated by Expression 2 described later in this specification. WithExpression 2, it is possible to define an evaluated value for a smallcorrelation window group, and it is possible to detect a movement of anobject by calculating a position of (xc′, yc′) at which the evaluatedvalue is minimized. Details of Expression 2 will be described later inthis specification.

Shown in FIG. 3 are a result of Comparative Example described above anda result of calculation of an evaluated value with correlation windowsbeing moved in the X-axis direction according to the method of thisinvention. In FIG. 3, a graph A1 shows the case of Comparative Example,and a graph A2 shows the case of this embodiment. The horizontal axisindicates the number of moved pixels, and the vertical axis indicatesthe evaluated value calculated by Expression 1. The evaluated value isnormalized by setting the maximum value to 1. As is apparent from thegraphs, the evaluated value is 0 when the number of moved pixels is 4 ineach of the cases, and detection subject movement amounts detected bythe methods are identical to each other (movement amount (number ofpixels)=4).

In terms of the shapes of graphs, the peak in the graph of thisembodiment is sharper than that of the graph of Comparative Example. Itis considered that the image actually picked up includes various noisesdue to an optical fluctuation, an electrical fluctuation, distortion dueto a lens, and the like. Such noises influence on the calculation ofevaluated value, and it is assumed that the noises are included in thegraphs shown in FIG. 3 in an operation result based on an actual image.When such noises are overlapped with a graph, it is considered that apeak position becomes indefinite in the case of a gentle peak while itis assumed that the peak position does not become indefinite when anamount of the noises is not too large in the case of a sharp peak. Thatis, according to the method of this embodiment, it is possible toaccurately detect a movement amount of a detection subject when somenoises are included.

(Flow of Movement Detection Processing)

Hereinafter, a flow of a movement detection processing in thisembodiment will be described with reference to a flowchart shown in FIG.4. In Step 1 (hereinafter referred to as S1), an image pickup processingon a region including a detection subject by the image pickup unit 2 isstarted. The image pick up processing is a moving image pickup. Imagedata picked up by the image pickup unit 2 are stored in the imagestorage unit 21 in the order of frames based on the control by thepicked-up image storage control unit 11.

Next, the detection subject region setting unit 12 takes out one offrames of the moving image data stored in the image storage unit 21 as astandard image and extracts at least one detection subject from thestandard image (S2). A typical example of a method for extractingdetection subject includes a template matching method. The templatematching method is an image processing method for assuming a position,an angle, a scale, and the like of an image corresponding to thetemplate image based on a prepared template image. The detection subjectregion setting unit 12 in this embodiment performs the detection subjectextraction by the template matching method, but the detection subjectextraction may be performed by any other method.

Next, the detection subject region setting unit 12 sets a detectionsubject region including an image region extracted as the detectionsubject (S3). In this embodiment, an outer rim of the detection subjectregion is set to a position that is outside from an outer rim of theimage region extracted as the detection subject by several pixels (2 to6 pixels, for example). Also, the thus-set detection subject region isformed into a rectangular region having M pixels in the longitudinaldirection and N pixels in the horizontal direction (each of M and N isan integer). In the case of extracting the detection subject by thetemplate matching method as described above, the shape of the detectionsubject region may be set in accordance with the prepared templateimage.

The set shape information of the detection subject region is stored inthe detection subject region storage unit 22. Though the detectionsubject region is the longitudinal region in this embodiment, the shapeis not limited thereto and may be an arbitrary one depending on theshape of the detection subject.

Next, the correlation window setting unit 13 extracts pluralcharacteristic points from the detection subject region including theimage region of the detection subject (S4). After that, the correlationwindow setting unit 13 sets a correlation window for the extractedcharacteristic points and a standard point (S5). Details of thecharacteristic points extraction processing and the processing forsetting the correlation windows and the standard point will be describedlater in this specification.

Next, the movement detection unit 14 obtains image data of the standardimage and image data of the reference image corresponding to the nextframe of the standard image from the image storage unit 21 (S6). Afterthat, the movement detection unit 14 performs an evaluation operationusing the correlation windows and the standard point set in S5 to detecta movement of the detection subject by determining corresponding point(point corresponding to the standard point) with which the evaluatedvalue is minimized and smaller than a predetermined threshold value onthe reference image (S7). Details of the movement detection processingwill be described later in this specification.

Next, the output control unit 15 performs a processing of outputting thereference image stored in the image storage unit 21 and informationindicating the movement of the detection subject detected by themovement detection unit 14 (S8). Specifically, the output control unit15 performs control for causing the display unit 4 to display themovement detection result. The display of the movement detection resultis performed in such a manner that a graphic enabling to identify thedetection subject is displayed and moved along the movement of thedetection subject or a movement locus of a specific part of thedetection subject is displayed.

After that, whether or not the processing is to be terminated is judgedin S9, and, when it is judged that the processing is not terminated yet,the reference image at the time of the judgment is set as the standardimage, and the corresponding point detected in S7 is set as the standardpoint to start processing from S6 again by setting the next frame as thereference image (S10).

Though the frame subsequent to the standard image is set as thereference image in the above example, the reference image is not limitedthereto and may be set at a predetermined frame interval.

(Characteristic Point Extraction Processing and Correlation WindowSetting Processing)

Hereinafter, the details of the characteristic point extractionprocessing and the correlation window setting processing performed bythe correlation window setting unit 13 will be described. Shown in FIG.5 are a detection subject B2 which is one example of image of thedetection subject and one example of detection subject region B1 set bythe detection subject region setting unit 12. In the examples shown inFIG. 5, the detection subject region B1 is a longitudinal region havingM pixels in the vertical direction and N pixels in the horizontaldirection (each of M and N is an integer). Also, upper and lower sidesof an outer rim of the detection subject region B1 is outside of upperand lower ends of an outer rim of the detection subject B2 by 2 pixels,and left and right sides of the outer rim of the detection subjectregion B1 are outside of left and right ends of the outer rim of thedetection subject B2 by 3 pixels.

Inside the detection subject region B1 set as described above, thecorrelation window setting unit 13 extracts the plural characteristicpoints from an image region of the detection subject B2. Points thatcharacterizing a partial shape of the detection subject B2 maypreferably be selected as the characteristic points. By thus selectingthe characteristic points, it is possible to accurately associate thedetection subject B2 in the standard image to the detection subject B2in the reference image in the movement detection processing.

Also, it is preferable that a contrast between the characteristic pointsand the surrounding pixels is high. When the contrast between thecharacteristic points and the surrounding pixels is high, it is possibleto ensure a difference in pixel value between the characteristic pointsand the surrounding pixels when some noises are generated as well as tosuppress the characteristic points from being hidden by the noises.

It is preferable that the characteristic points are selected evenly fromthe entire part of the image region of the detection subject B2. Whenthe characteristic points are selected from one part of the image regionof the detection subject B2, the association is not based on the entireshape of the detection subject B2, thereby increasing possibility of afailure in the association processing.

In view of the foregoing, the characteristic points are selected asfollows in this embodiment. The correlation window setting unit 13creates four filters for extracting four corners, i.e. an upper leftcorner, an upper right corner, a lower left corner, and lower rightcorner, within the shape of the detection subject B2. The filters arereferred to as the first filter F1, the second filter F2, the thirdfilter F3, and the fourth filter F4. In the case where it is unnecessaryto discriminate the first to forth filters F1 to F4 from one another,each of the filters is referred to as the filter F. In the case ofextracting the detection subject by the template matching method, thefirst to fourth filters F1 to F4 are created corresponding to thetemplate image in advance of the characteristic point extraction.

Each of the filters F is in a matrix arrangement of m×n (each of m and nis an integer; m<M and n<N), and a value of each of elements is set to 0or 1 depending on the detection shape. By applying the filters F to thepixels of the detection subject region B1 to calculate a value which isa sum of absolute values of differences between the values of theelements of the filters F and the values of the pixels of thecorresponding detection subject region B1 as a filtering result. Withsuch filtering processing, the filtering result values of the pixelscorresponding to the characteristic points to be detected by the filtersF becomes lower than values of other pixels.

Shown in FIG. 6( b) is the matrix arrangement of the first filter F1 andthe result obtained by applying the first filter F1 to the detectionsubject region B1; shown in FIG. 6( c) is the matrix arrangement of thesecond filter F2 and the result obtained by applying the second filterF2 to the detection subject region B1; shown in FIG. 6( d) is the matrixarrangement of the third filter F3 and the result obtained by applyingthe third filter F3 to the detection subject region B1; and shown inFIG. 6( e) is the matrix arrangement of the fourth filter F4 and theresult obtained by applying the fourth filter F4 to the detectionsubject region B1.

Shown in FIG. 6( a) is a result of selecting a minimum value of thefiltering result values in the processing results shown in FIGS. 6( b)to 6(e) for the pixels. As shown in FIG. 6( a), the filtering resultvalues corresponding to the upper left, upper right, lower left, andlower right four corners of the detection subject B2 are 0, and thesefour pixels are selected as the characteristic points.

Note that it is considered that the filtering result value to beselected as the characteristic point may not be 0 since the ideal pixelvalues shown in FIG. 5 are not achieved in an actual image due to thenoises included in the actual image. Therefore, a filtering resultthreshold value Cth for the filtering result value to be selected as thecharacteristic point is set in an actual processing, and a pixelachieving a filtering result value smaller than the filtering resultthreshold value Cth is selected as the characteristic point.

The size of the correlation window is set to be the same as that of thematrix of the filter F. The size of the correlation window and the sizeof the filter F may be a constant value or may be varied depending onthe size of the detection subject region B1. In the case of varying thesizes depending on the size of the detection subject region B1, each ofthe sizes of the correlation window and the filter F may be set inaccordance with a proportion of the size to the size of the detectionsubject region B1, such as (M/4)×(N/4) when the size of the detectionsubject region is represented by M×N.

In the case where a large number of pixels has the filtering resultvalue smaller than the filtering result threshold value Cth, all ofthese pixels may be selected as the characteristic points, but thenumber of pixels requiring the association in the movement detectionprocessing is increased when the entire pixels are selected as thecharacteristic point, thereby undesirably increasing the processingtime. Therefore, it is preferable to perform thinning of thecharacteristic points as described below.

An upper limit value Nth for the number of characteristic points is setin advance of the thinning processing. The upper limit value Nth is setfor the purpose of preventing the processing time to becomeunnecessarily long. When the number of pixels each having the filteringresult value smaller than the filtering result threshold value Cthexceeds the upper limit value Nth, the thinning processing is performedbased on the following standards. As the first standard, a method ofselecting Nth characteristic points in an ascending order of filteringresult values is employed. As the second standard, a method of selectingNth characteristic points in an ascending order of distances from thecorners of the outer rim of the detection subject region B1 is employed.As the third standard, Nth characteristic points are selected in anascending order of distances from the sides of the outer rim of thedetection subject region B1.

According to the first standard, since the pixels in the pixel regionscloser to the filters F for detecting the characteristic points areselected as the characteristic points, it is possible to more accuratelyperform the association in the movement detection processing. Accordingto the second and third standards, since the pixels closer to the outerrim of the detection subject region B1 are selected as thecharacteristic points, it is possible to enhance the possibility for thecharacteristic points to be selected from the entire image region of thedetection subject B2, thereby enabling to more accurately perform theassociation due to the realization of the association based on theentire shape of the detection subject B2.

Though the sizes of the filter F and the correlation window are set tothe identical value in the above example, the sizes may be differentfrom each other.

(Movement Detection Processing)

Hereinafter, the movement detection processing by the movement detectionunit 14 will be described. The movement detection unit 14 obtains theimage data of the standard image and the image data of the referenceimage and performs the evaluation operation based on the correlationwindows set by the correlation window setting unit 13. In this case, itis assumed that the Nth characteristic points have been selected by thecorrelation window setting unit 13, and that coordinates of the selectedcharacteristic points are defined as (x0, y0), (x1, y1), (x2, y2), . . .(xNth−1, yNth−1). The following operation using an evaluation functionis performed for the pixels of the reference image based on expansion ofexpression using SAD (Sum of Absolute Difference).

[Expression 2]

In Expression 2, (xc, yc) indicates a central coordinate (standardpoint) of the detection subject on the standard image as shown in FIG.7( a). The central coordinate of the detection subject is a barycentricposition of the selected plural characteristic points. That is, (xc, yx)is represented by the following expression.

[Expression 3]

Relative coordinates for the central coordinate of the points arerepresented by the following expression.

[Expression 4]

As shown in FIG. 7( b), (x′c, y′c) indicates a central coordinate of thedetection subject on the reference image. In this case, (x′c, y′c) isset in a range (search range: P×Q pixel region (each of P and Q is aninteger) in which the detection subject can move.

The search range is set as follows. When the detection subject is a car,for example, and the car travels on a specific road, the detectionsubject travels in one direction unless the car is at an intersection.Also, by using a time interval which is correlative to an assumed speedof the detection subject (200 km/h for a car, for example), it ispossible to detect a distance that the detection subject moves in acertain period. By combining the travel direction information and themovable distance information, it is possible to set the search range as“the range in the detection subject can move”.

A pixel value in each of the coordinates of the correlation windows ofthe standard image is represented by k, and a pixel value in each ofcoordinates in the correlation windows of the reference image isrepresented by k′. A horizontal pixel number of each of the setcorrelation windows is represented by m, and a vertical pixel number ofeach of the set correlation windows is represented by n. In the casewhere each of m and n is an odd number, a value such as m/2 isappropriately converted into an integer.

The movement detection unit 14 considers a pixel position on thereference image at which the evaluated value E (x′c, y′c) is minimizedas the central coordinate of the detection subject on the referenceimage. That is, a difference between the central coordinate of thedetection subject on the standard image and the central coordinate ofthe detection subject on the reference image is considered to be amovement amount of the detection subject.

Though barycentric coordinate of the plural characteristic points isused as the standard point in the above example, the standard point isnot limited to the example, and any position may be set as the standardpoint insofar as the position is determined unambiguously based on theselected characteristic points. As such standard point, onecharacteristic point selected among the plural characteristic pointsbased on a predetermined standard, a midpoint of two characteristicpoints selected based on a predetermined standard, or the like may beused.

Though the processing is performed by assuming the image data as dataformed from one color component, such as a monochrome image, in theabove example, the image data is not limited to the monochrome image,and the movement detection processing may be performed on colorcomponents by using data formed from three color components of colors ofRGB. However, since the processing time is increased for the pluralcolors in the case of performing the processing on plural colorcomponents, it is preferable to perform the processing on the dataformed from one color component when it is possible to perform themovement detection based on one color component. Also, in the case wherethe image data are color image data, the above-described movementdetection processing may be performed by comparing colors of the pixels.

(Processing in the Case where Size of Detection Subject Changes)

In the case where a movement of a detection subject occurs in such amanner that a distance between the detection subject and the imagepickup unit 2 changes, the size of the detection subject in the imageregion changes on an image picked up by the image pickup unit 2. Shownin FIG. 8 is a state in which the detection subject that has existed ata position A moves to a position B in a direction approaching to theimage pickup unit 2. In this case, the size of the detection subject atthe position B is larger than the size of the detection subject at theposition A.

Hereinafter, a case of setting a region including an entire image regionof a detection subject as a correlation window will be described asComparative Example. In this case, as show in FIGS. 9( a) and 9(b), thecorrelation window in the standard image and the correlation window inthe reference image differs in size. When the standard image and thereference image differ from each other in correlation window size, it isimpossible to perform the above-described movement detection using theevaluation factor. Therefore, in this case, the movement detection unit14 performs a processing of enlarging the image in the correlationwindow M×N in the standard image to the size of the correlation windowin the reference image, i.e. to the image of M′×N′. Since the numbers ofpixels included in the standard image and the reference image becomesthe same by the enlargement processing, it is possible to perform themovement detection using the evaluation factor in this state.

However, in this Comparative Example, there are problems that a time forthe image enlargement processing is required and that the evaluationoperation processing time is increased due to the increase in pixelnumber. Though it is possible to set the numbers of the pixels to anidentical value by reducing the size of the correlation window in thereference image to the size of the correlation window in the standardimage, the size of the correlation window is relatively large, and theprocessing time is still long.

In contrast, by performing the following processing by the movementdetection unit 14 in this embodiment, it is possible to deal with theenlargement/reduction of a detection subject with the sizes of thecorrelation windows of the standard image and the reference image beingthe same.

As shown in Expressions 2 and 4, in the case of performing the operationof the evaluation factor, each of the coordinates in the correlationwindows is represented by a relative position from the centralcoordinate (standard point) of the detection subject. Therefore, inorder to deal with the enlargement/reduction of the detection subject, alength corresponding to the relative coordinate of the centralcoordinate (xc, yc) of the detection subject and each of the centralcoordinates of the correlation windows (xci, yxi) (i=0, 1, 2, . . .Nth−1) is enlarged or reduced as shown in FIGS. 10( a) and 10(b). Thatis, when an enlargement reduction ratio is K, it is possible to dealwith the enlargement/reduction of the detection subject only by changingthe relative coordinates as described in the following expression withthe sizes of the correlation windows of the standard image and thereference image being the same. (xci, yci)=K(xci, yci), wherein i=0, 1,2, . . . . Nth−1

After converting the relative coordinates as described above, anevaluated value is obtained by using the evaluation factor representedby Expression 2. With such processing, it is sufficient to convert therelative position between the correlation windows and the centralcoordinate of the detection subject, and it is unnecessary to change thesize of the correlation window. That is, since the number of pixels tobe processed is not increased when the image region of the detectionsubject is enlarged due to the approach of the detection subject, it ispossible to deal with the enlargement without causing the increase inprocessing time.

(Determination of Enlargement/Reduction Ratio)

Basically, a value of the enlargement/reduction ratio K is not decideduniquely and varies largely depending on the size of the picked-up imageof the detection subject. Therefore, in actuality, several types ofenlargement/reduction ratios are given to perform the evaluationoperation so that pixels that achieve the best correlativity (pixelshaving evaluation values nearly equal to 0) are ultimately selected forperforming the movement detection. That is, in order to reduce theprocessing time, it is preferable to narrow down a range of values ofthe enlargement/reduction ratios K.

The size of the detection subject on the picked-up image is inverselyproportional to the distance between the detection subject and the imagepickup unit 2. That is, in the case where the distance between thedetection subject and the image pickup unit 2 has changed from L1 to L2due to a movement of the detection subject, the value of theenlargement/reduction ratio K becomes L1/L2. Therefore, when it ispossible to detect the distance between the detection subject and theimage pickup unit 2, it is possible to uniquely decide the value of theenlargement/reduction ratio K.

For example, in the case where the image pickup unit 2 is disposed asbeing fixed with respect to a detection subject moving on a plane S asshown in FIG. 11, the movement detection unit 14 calculates a distancebetween the detection subject and the image pickup unit 2 based on aposition of the detection subject on a picked-up image. That is, when anangle formed by a straight line connecting a Y coordinate y of aprojection position of the detection subject in the image pickup element2A provided in the image pickup unit 2 to the center of the image pickuplens 2B provided in the image pickup unit 2 and an optical axis of theimage pickup lens 2B is represented by φ, the distance L between theimage pickup unit 2 and the detection subject is represented by thefollowing expression.

[Expression 5]

In Expression 5, Hc represents a height at the center of the imagepickup lens in a direction of a normal line with respect to the plane S;Hw represents a height of the detection subject in a direction of anormal line with respect to the plane S; η represents an angle formed bythe optical axis of the image pickup lens 2B and the plane S; and frepresents an optical distance between the image pickup lens 2B and theimage pickup element 2A.

That is, since it is possible to detect the Y-coordinate y on the imagepickup element 2A from the picked-up image, it is possible to calculatethe distance between the image pickup unit 2 and the detection subjectaccording to Expression 5. Therefore, in the case where the image pickupunit 2 is fixedly disposed and the detection subject moves in thetwo-dimensional plane, it is possible to uniquely decide the value ofthe enlargement/reduction ratio K depending on the position to whicheach of the correlation windows is applied in the reference image.

(Determination of Enlargement/Reduction Ratio by Stereo Image)

In the above example, the value of the enlargement/reduction ratio K isuniquely decided on condition that the detection subject moves in thetwo-dimensional plane. However, in the case where the movement range ofthe detection subject moves in a three-dimensional space in anunspecified manner, it is impossible to uniquely decide the value of theenlargement/reduction ratio K since the value of Hw in Expression 5changes.

As a structure for dealing with such case, a structure wherein two imagepickup units 2 are provided as shown in FIG. 12 is provided. Byproviding two image pickup unit 2 and picking up the identical detectionsubject B2 by each of the image pickup units 2, it is possible to obtaintwo images of the detection subject B2 which are picked up fromdifferent angles. It is possible to uniquely decide the value of theenlargement/reduction ratio K by calculating distances between the imagepickup units 2 and the detection subject B2 by using the movementdetection unit 14 based on the thus-obtained stereo images.

Hereinafter, specific description will be given based on FIG. 12. InFIG. 12, B indicates a distance between the centers of the image pickuplenses 2B of the two image pickup units 2. L indicates a distancebetween the detection subject B2 and the two image pickup units 2. To bestrict, a distance between one the image pickup unit 2 and the detectionsubject B2 differs from a distance between the other image pickup unit 2and the detection subject B2. However, since the value of B with respectL is remarkably small in actuality, the distances between the imagepickup units 2 and the detection subject 2 are approximated to berepresented by L. An optical distance between the image pickup lens 2Band the image pickup element 2A in each of the image pickup units 2 isrepresented by f.

When a distance between an image location of the detection subject onthe image pickup element 2A in one of the image pickup units 2 and animage location of the detection subject on the image pickup element 2Ain the other image pickup unit 2 is set as a disparity d, L is detectedby the following equation:L=B×f/d.

In the case where: a disparity in a central coordinate (xc, yc) of thedetection subject B2 of the standard image is represented by dc; adistance is represented by Lc; a disparity in a central coordinate (x′c,y′c) of the detection subject B2 of the reference image is representedby dc′; and a distance is represented by Lc′, the enlargement/reductionratio K is decided by the following equation in view of the fact that animage pickup size of the detection subject is inversely proportional tothe distance:K=Lc/Lc′.

Note that the enlargement/reduction ratio k is generally calculated byusing a disparity directly derived from the image processing in theactual processing. That is, as described in the foregoing, since thedistance and the disparity are in the inverse proportion relationship,the enlargement/reduction ratio K may be decided by the followingequation:K=dc′/dc.

In the case of detecting the disparity or the distance based on thestereo images as described above, it is necessary to associate thedetection subjects included in left images of the stereo images witheach other as well as to associate the detection subjects included inright images of the stereo images with each other. It is possible toperform the detection subject association in the stereo images by amethod of similarity judgment in the detection subject images or thelike.

(Case Wherein Detection Subject Rotates)

Shown in FIG. 13( a) is a state of picking up images in the case wherethe detection subject is a car and when the car approaches to the imagepickup unit 2 while rounding a curve. Shown in FIG. 13( b) is a statewherein the image of 13(a) is displayed as a binary image. In thesedrawings, the images indicated by “PRE-PREVIOUS”, “PREVIOUS”, “CURRENT”are an image of a frame precedent to the current image by two frames, animage of a frame precedent to the current image by one frame, and animage of a current frame. As shown in FIG. 13( b), four corners of thecar are selected as characteristic points based on the pre-previousimage of the detection subject, and a correlation window is providedaround each of the characteristic points.

The car travels almost straight in the pre-previous image and theprevious image, and there is little change in orientation (posture) ofthe car from the pre-previous image to the previous image. Therefore,the pre-previous image of the detection subject and the previous imageof the detection subject are substantially similar to each other.Accordingly, it is possible to associate the detection subject in thepre-previous image with the detection subject in the previous imageaccurately by setting a value of the enlargement/reduction ratio K bythe above-described method.

In contrast, since the orientation of the car of the current image isdifferent from that of the previous image, it is difficult to accuratelyperform the association simply by enlarging/reducing the distancebetween the correlation windows and the standard point. The reason forthe difficulty is that an orientation of characteristic points withrespect to a central coordinate of the detection subject of the previousimage is different from that of the current image.

Accordingly, the movement detection unit 14 performs a rotationconversion indicated by the following expression on the coordinates ofthe characteristic points to perform the association by rotating thecoordinates of the characteristic points with respect to the centralcoordinate of the detection subject.

[Expression 6]

In Expression 6, θ represents a rotation angle. Theenlargement/reduction ratio is represented by K as in the foregoingexamples. With such conversion, it is possible to deal with the changein orientation and the change in size of the detection subject inpicked-up images.

(Determination of Rotation Angle)

Basically, the value of the rotation angle θ is not decided uniquely andvaries largely depending on the change in orientation of the detectionsubject. Therefore, in actuality, the movement detection unit 14 givesseveral types of rotation angle θ to perform the evaluation operation,and the pixels (pixels each having an evaluated value close to 0) thatachieve the best correlativity are ultimately selected for performingthe movement detection. That is, in order to reduce the processing time,it is preferable to narrow down a range of values of the rotation angleθ. Therefore, as described below, in the case where it is possible todetermine an orientation of a detection subject depending on a region ofa picked-up image, it is possible to reduce the processing time byuniquely deciding the value of the rotation angle θ by using theorientation.

Shown in FIG. 14( a) is one example of the case wherein the detectionsubject is a car traveling on a road having a branch point. In suchcase, an orientation of the car is decided basically depending on adirection of the road on which the car travels.

Shown in FIG. 14( b) is a region of the road included in the picked-upimage. As shown in FIG. 14( b), a region 1 which is a straight roadregion and a region 2 which is a straight road region exist, and it ispossible to uniquely decide the orientation of the car in each of theregions. That is, when an angle formed by the direction of the road ofthe region 1 and the direction of the road of the region 2 is θ, therotation conversion by Expression 6 using the determined θ is performedfor association in the case where the car moves from the region 2 to theregion 1.

Thus, in the case where it is possible to identify the condition relatedto the orientation of the detection subject by the position of thedetection subject in the picked-up image, it is possible to uniquelydecide the value of the rotation angle θ depending on the position towhich the movement detection unit 14 applies the correlation window inthe reference image by setting a rotation relationship between theregions in the picked-up image in advance of the movement detection.

(Rotation of Correlation Window)

Shown in FIG. 15 is an enlarged view of one correlation window in theprevious and current images of the detection subject shown in FIG. 13(b). As shown in FIG. 15, when the orientation of the detection subjectchanges, an orientation of the image of the correlation window changes.Accordingly, influence exerted on an evaluated value to be obtained bythe evaluation function is increased with an increase in angle change inorientation of the detection subject, thereby increasing difficulty inperforming association.

Therefore, in order to deal with the above case, the movement detectionunit 14 not only converts the relative coordinate of the characteristicpoints as described above but also rotates the images in the range ofthe correlation windows about the central coordinate of each of thecorrelation windows. By performing such processing, it is possible toperform the association of the detection subject of the standard imagewith the detection subject in the reference image in the case where theorientation of the detection subject largely changes.

Due to the addition of the rotation conversion processing for the imagesin the correlation windows, the amount of operation is more or lessincreased. However, since the number of pixels of the images in thecorrelation windows is small, the processing time is not increasedprominently.

(Specific Examples of Setting Characteristic Points and CorrelationWindows)

Hereinafter, description will be given on specific examples of theprocessing for extracting the characteristic points and the processingfor setting correlation windows that are performed by the correlationwindow setting unit 13 as described in the foregoing with reference toFIGS. 6( a) to 6(d). Shown in FIG. 16( a) is one example of thedetection subject region B1 including the detection subject region B2 ina state where a car is picked up as the detection subject B2. When thefiltering processing is performed on the detection subject region B1 byusing the filters shown in FIG. 17, plural pixels achieving filteringresult values that are less than a predetermined threshold value areselected. In FIG. 16( a), pixels indicated by x are the selected pixels.A contrast between the thus-obtained selected points and the surroundingpixels is relatively high.

Though the filters shown in FIG. 17 are those that detect an edge in ahorizontal direction, it is preferable to appropriately set an optimumfilter depending on the orientation of image pickup of the detectionsubject, the shape of the detection subject, and the like.

After that, the detection subject region B1 is divided into pluralpartial regions (intermediate regions) as shown in FIG. 16( b).Selection points having a high contrast are selected one by onesequentially from the partial regions, and the selection of the selectedpoints from the partial regions is repeated until the number of theselected points reaches to a predetermined value, thereby setting thecharacteristic points (FIG. 16( c)). When the characteristic points areselected as described above, the characteristic points are selectedevenly from the partial regions. By selecting the characteristic pointsevenly from the partial regions, it is possible to perform theassociation based on the entire shape of the detection subject B2. Afterthat, a correlation window is set around each of the selectedcharacteristic points as shown in FIG. 16( d).

In the case of detecting a movement of a person as a detection subject,filters shown in FIG. 18 may be used. In this case, it is assumed that aheat portion and a foot portion of a person as the detection subject areincluded in the picked-up image. Among four filters shown in the lefthand side of FIG. 18, upper two filters are filters for detecting anupper left an upper right of the head portion, and lower two filters arefilters for detecting a lower left and a lower right of the footportion. The filters may be used as they are, and images of the filtersmay be simplified depending on the sizes of the correlation windows asshown on the right hand side of FIG. 18.

When the filtering processing is performed by using the above filtersfor the detection subject region B1, plural pixels each achieving afiltering result lower than the threshold value are selected. As shownin FIG. 19, the detection subject region B1 is divided into pluralpartial regions. In the same manner as described above, a selected pointhaving a high contrast is selected from each of the partial regionssequentially, and the selection of the selected points from the partialregions is repeated until the number of the selected points reaches to apredetermined value to select the characteristic points.

(Another Example of Flow of Movement Detection Processing)

In the flowchart of the movement detection processing shown in FIG. 4,steps similar to the steps from S2 to S5 may be performed on thereference image after S6. FIG. 20 is a flowchart showing a flow ofmovement detection processing in such case. Since steps of S21 to S25are the same as those of S1 to S5 in FIG. 4, description for S21 to S25is omitted.

In S26, the detection subject region setting unit 12 obtains image dataof the reference image corresponding to the next frame of the standardimage from the image storage unit 21 and extracts at least one detectionsubject from the reference image in the same manner as in the processingin S2. After that, in the same manner as in the processing of S3 to S5,processing of S27 to S29 is performed. That is, the detection subjectregion setting unit 12 sets the detection subject region in S27, thecorrelation window setting unit 13 extracts plural characteristic pointsin S28, and the correlation window setting unit 13 sets correlationwindows and a standard point in S29.

After that, the movement detection unit 14 obtains image data of thestandard image and image data of the reference image corresponding tothe next frame of the standard image from the image storage unit 21(S30). After that, the movement detection unit 14 performs theevaluation operation using the correlation windows and the standardpoint set in S25 to detect a movement of the detection subject bydetermining a corresponding point with which the evaluated value isminimized and smaller than a predetermined threshold value on thereference image (S31). What is different from the processing of S7 isthat the search range for searching the corresponding point in thereference image is changed to the standard point set in S29. That is,the movement of the detection subject is detected by determining thedetection subject that is judged to be identical with the detectionsubject determined in the standard image among the detection subjectsdetermined in the reference image. The subsequent processing of S32 toS34 is the same as the processing of S8 to S10 in FIG. 4.

The above-described processing is suitable for traffic surveillance andpasserby surveillance wherein many detection subjects move in and out animage pickup range. In this case, a multiple of detection subjects areextracted from the standard image (N detection subjects from B1 to BN,for example), and correlation windows and a standard point are set foreach of the detection subjects in S22. Also, M detection subjects (B′1to B′M) are extracted from the reference image, and correlation windowsand a standard point are set for each of the detection subjects in S26.

After that, in S31, the standard points (xmc′, ymc′) (m=1 to M) of thedetection subjects B′1 to B′M on the reference image are assigned toExpression 2 for the standard points (xnc, ync) (n=1 to N) of thedetection subjects B1 to BN on the standard image, and a pair achievinga smallest evaluated value E which is smaller than a predeterminedthreshold value is associated to each other as an identical object.

In the case where it is impossible to associate any of the detectionsubjects B1 to BN, such detection subject is treated as having beenexited from the image pickup range. In the case where it is impossibleto associate any of the detection subjects B′1 to B′M, such detectionsubject is treated as having been entered into the image pickup range.

That is, according to the above-described processing, it is possible totrack an object that does not exist in the standard image but existsonly in the reference image.

Also, according to the above-described processing, since the search forthe corresponding points is limited to the standard points of thedetection subjects determined in the reference image, it is possible toperform the movement detection at high speed and with accuracy ascompared to the case of searching the entire points in the referenceimage.

In the case where the detection subjects are detected in the standardimage as shown in FIG. 21( a) and the detection subjects are detected inthe reference image as shown in FIG. 21( b), movement tracking for eachof the detection subjects is realized as shown in FIG. 21( c) accordingto the above-described processing.

(Flow of Movement Detection Processing when Using Stereo Images)

Hereinafter, a flow of movement detection processing in a structure ofdetermining an enlargement/reduction ratio by stereo images will bedescribed. In the case of determining the enlargement/reduction ratio bystereo images, it is necessary to calculate a disparity (distance) ofeach of the standard points of the detection subjects in both of thestandard image and the reference image. Therefore, as in the processingof the flowchart shown in FIG. 20 described above, it is necessary todetermine the detection subjects in both of the standard image and thereference image. Also, the determination of detection subject isperformed for each of the stereo images of the standard image and thereference image.

Specifically, the following processing is performed. In S31 of FIG. 20,the movement detection unit 14 calculates disparities d1 c to dNc (ordistances L1 c to LNc) by the above-described method for the standardpoints (xnc, ync) (n=1 to N) of the detection subjects B1 to BN in thestandard image. Also, in the same manner, the movement detection unit 14calculates disparities d1 c′ to dMc′ (or distances L1 c′ to LNc′) by theabove-described method for the standard points (xmc′, ymc′) (m=1 to M)of the detection subjects B′1 to B′M in the reference image.

Next, the movement detection units 14 calculates anenlargement/reduction ratio K by the above-described expression for thecombinations of the standard points in the standard image and thestandard points in the reference image by setting K=dmc′/dnc (orK=Lnc/Lmc′). After that, the movement detection unit 14 assigns theenlargement/reduction ratio K to the above-described expression of (xci,yci)=K(xci, yci) to convert a relative coordinate and detects anevaluated value by using the evaluation function shown in Expression 2.As described above, the combination of the standard point and thecorresponding point that achieves the smallest whose evaluated value Eis the smallest (and not more than a predetermined threshold value) isassociated to each other as an identical object to detect the movementof the object.

According to the above processing, since the enlargement/reduction ratioK is narrowed down to a certain degree, it is possible to reduce theprocessing time.

(Specific Structure of Image Processing Apparatus)

The blocks provided in the image processing unit 3, particularly, thepicked-up image storage control unit 11, the detection subject regionsetting unit 12, the correlation window setting unit 13, the movementdetection unit 14, and the output control unit 15 may be structured by ahardware logic or may be realized by software by using a CPU.

That is, the image processing unit 3 is provided with a CPU (centralprocessing unit) for executing instructions of a control programrealizing the functions, a ROM (read only memory) storing the program,and a RAM (random access memory) developing the program, a memory device(recording medium) such as a memory for storing the program and variousdata, and the like. The object of this invention is also attained bysupplying the recording medium recording program cords (an executionformat program, an intermediate cord program, a source program) of thecontrol program which is the software realizing the functions of theimage processing unit 3 in such a fashion that a computer can read andby causing the computer (CPU or MPU) to read out and execute the programcords recorded on the recording medium.

Usable as the recording medium are tapes such as a magnetic tape and acassette tape; disks including magnetic disks such as a floppy disk(trade name) and a hard disk and optical disks such as a C-ROM, an MO,an MD, a DVD, and a CD-R; cards such as an IC card (including memorycard) and an optical card; semiconductor memories such as a mask ROM, anEPROM, an EEPROM, and a flash ROM; and the like.

Also, the image processing unit 3 may have a structure that is capableof accessing a communication network so that the program code issupplied via the communication network. The communication network is notparticularly limited, and usable communication networks are theinternet, an intranet, an extranet, a LAN, an ISDN, a VAN, a CATVcommunication network, a virtual private network, a telephone linenetwork, a mobile communication network, a satellite communicationnetwork, and the like. Also, a transmission medium for structuring thecommunication network is not particularly limited, and wiredtransmission such as IEEE1394, an USB, a electric power line carrier, acable TV line, a telephone line, and an ADSL line and a wirelesstransmission such as an infrared ray including IrDA and a remotecontroller, the Bluetooth (trade name), 802.11 wireless communication,HDR, a mobile phone network, a satellite circuit, and a ground wavedigital network are usable. This invention can be realized in anembodiment of computer data signals embedded in a carrier wave, which isan embodiment of the program code by an electronic transmission.

This invention is not limited to the embodiments described above, and itis possible to make various alterations in the scope of claims. That is,embodiments obtained by combining technical means that are appropriatelyaltered within the scope of claims are encompassed by the technicalscope of this invention.

INDUSTRIAL APPLICABILITY

It is possible to use the moving object tracking system according tothis invention for traffic control, vehicle travel assistance, and thelike by determining movements of vehicles. Also, it is possible to usethe moving object tracking system for passersby status investigation,customer behavior investigation, and the like by determining movementsof passersby in public areas and movements of customers and the like inretail premises. Also, the moving object tracking system is usable forproduction control, production planning, and the like by determiningmovements of production subject in a production line in a plant andmovements of workers.

1. An image processing apparatus comprising: a CPU configured to: obtainan acquired image at a first time point as a standard image fromtime-series acquired image data obtained by an image acquisition unitacquiring a tracking subject and select a plurality of partial imageregions including an image region of the tracking subject in thestandard image, each of the plurality of partial image regions beingseparately spaced from the other of the plurality of partial imageregions; obtain an acquired image at a second time point that isdifferent from the first time point as a reference image and detect amovement of the tracking subject between the standard image and thereference image based on a difference between a state of pixel values ofthe partial image regions of the standard image and a state of pixelvalues of partial image regions of the reference image; set a center ofeach of the plurality of partial image regions as a standard point usedas a standard for relative positions of the partial image regions; andobtain the state of the pixel values of the partial image regions of thereference image with the relative positions being unchanged, therelative positions being the relative positions of the partial imageregions with respect to the standard point, and calculate the differencebetween the obtained state of the pixel values of the partial imageregions and the state of the pixel values of the partial image regionsof the standard image to calculate a position of the standard point inthe reference image.
 2. The image processing apparatus according toclaim 1, wherein a total of numbers of pixels in the plural partialimage regions is smaller than the number of pixels of the image regionof the tracking subject.
 3. The image processing apparatus according toclaim 1, wherein the is further configured to divide the image region ofthe tracking subject into a plurality of intermediate regions to selectat least one of the intermediate regions as the partial image region. 4.The image processing apparatus according to claim 1, wherein the CPU isfurther configured to select a plurality of characteristic points byapplying a filter for detecting the characteristic points characterizinga partial shape of the tracking subject to the standard image to set aregion including the characteristic points as the partial image region.5. A moving object detection system comprising: an image acquisitionunit for acquiring a tracking subject and the image processing apparatusdefined in claim
 1. 6. An image processing apparatus comprising: a CPUconfigured to: obtain an acquired image at a first time point as astandard image from time-series acquired image data obtained by an imageacquisition unit acquiring a tracking subject and select a plurality ofpartial image regions including an image region of the tracking subjectin the standard image, each of the plurality of partial image regionsbeing separately spaced from the other of the plurality of partial imageregions; obtain an acquired image at a second time point that isdifferent from the first time point as a reference image and detect amovement of the tracking subject between the standard image and thereference image based on a difference between a state of pixel values ofthe partial image regions of the standard image and a state of pixelvalues of partial image regions of the reference image; set a center ofeach of the plurality of partial image regions as a standard point usedas a standard for relative positions of the partial image regions, therelative positions of the partial image regions of the standard imagebeing identical to respective of the relative positions of the partialimage regions of the reference image; and obtain the state of the pixelvalues of the partial image regions of the reference image by convertingthe relative positions of the partial image regions with respect to thestandard point by a predetermined conversion processing and calculate aposition of the standard point in the reference image by calculating thedifference between the obtained state of the pixel values of the partialimage regions and the state of the pixel values of the partial imageregions of the standard image.
 7. The image processing apparatusaccording to claim 6, wherein the CPU is further configured to perform aprocessing of enlarging or reducing a distance between each of therelative positions of the partial image regions and the standard pointas the predetermined conversion processing.
 8. The image processingapparatus according to claim 7, wherein the CPU is further configured todecide an enlargement/reduction ratio based on a position of thedetection subject in the standard image and a position of the trackingsubject in the reference image.
 9. The image processing apparatusaccording to claim 7, wherein the CPU is further configured to obtainfrom the image acquisition unit two acquired images obtained byacquiring the tracking subject from two different points simultaneouslyand calculate a distance between the tracking subject and the imageacquisition unit in the standard image and a distance between thetracking subject and the image acquisition unit in the reference imagebased on the two acquired images to decide the enlargement/reductionratio based on the distances.
 10. The image processing apparatusaccording to claim 6, wherein the CPU is further configured to performas the predetermined conversion processing a processing of rotating therelative positions of the partial image regions, which are relative tothe standard point.
 11. The image processing apparatus according toclaim 10, wherein the CPU is further configured to decide a rotationangle based on the position of the tracking subject in the standardimage and the position of the tracking subject in the reference image.12. The image processing apparatus according to claim 10, wherein theCPU is further configured to obtain the state of the pixel values of thepartial image regions by rotating the partial image regions of thereference image.
 13. An image processing method comprising: obtaining,via a processor, an acquired image at a first time point as a standardimage from time-series acquired image data obtained by an imageacquisition unit acquiring a tracking subject and selecting a pluralityof partial image regions including an image region of the trackingsubject in the standard image, each of the plurality of partial imageregions being separately spaced from the other of the plurality ofpartial image regions; obtaining, via the processor, an acquired imageat a second time point that is different from the first time point as areference image and detecting a movement of the tracking subject betweenthe standard image and the reference image based on a difference betweena state of pixel values of the partial image regions of the standardimage and a state of pixel values of partial image regions of thereference image; setting, via the processor, a center of each of theplurality of partial image regions as a standard point used as astandard for relative positions of the partial image regions; andobtaining, via the processor, the state of the pixel values of thepartial image regions of the reference image with the relative positionsbeing unchanged, the relative positions being the relative positions ofthe partial image regions with respect to the standard point, tocalculate the difference between the obtained state of the pixel valuesof the partial image regions and the state of the pixel values of thepartial image regions of the standard image.
 14. An image processingmethod comprising: obtaining, via a processor, an acquired image at afirst time point as a standard image from time-series acquired imagedata obtained by an image acquisition unit acquiring a tracking subjectand selecting a plurality of partial image regions including an imageregion of the tracking subject in the standard image, each of theplurality of partial image regions being separately spaced from theother of the plurality of partial image regions, and obtaining, via aprocessor, an acquired image at a second time point that is differentfrom the first time point as a reference image and detecting a movementof the tracking subject between the standard image and the referenceimage based on a difference between a state of pixel values of thepartial image regions of the standard image and a state of pixel valuesof partial image regions of the reference image; setting, via theprocessor, a center of each of the plurality of partial image regions asa standard point used as a standard for relative positions of thepartial image regions, the relative positions of the partial imageregions of the standard image being identical to respective of therelative positions of the partial image regions of the reference image;and obtaining, via the processor, the state of the pixel values of thepartial image regions of the reference image by converting the relativepositions of the partial image regions with respect to the standardpoint by a predetermined conversion processing to calculate a positionof the standard point in the reference image by calculating thedifference between the obtained state of the pixel values of the partialimage regions and the state of the pixel values of the partial imageregions of the standard image.
 15. A non-transitory recording mediumstoring an image processing program, the program, when executed by acomputer causing the computer to perform the steps of: obtaining anacquired image at a first time point as a standard image fromtime-series acquired image data obtained by an image acquisition unitacquiring a tracking subject and selecting a plurality of partial imageregions including an image region of the tracking subject in thestandard image, each of the plurality of partial image regions beingseparately spaced from the other of the plurality of partial imageregions; obtaining an acquired image at a second time point that isdifferent from the first time point as a reference image and detecting amovement of the tracking subject between the standard image and thereference image based on a difference between a state of pixel values ofthe partial image regions of the standard image and a state of pixelvalues of partial image regions of the reference image; setting a centerof each of the plurality of partial image regions as a standard pointused as a standard for relative positions of the partial image regions;and obtaining the state of the pixel values of the partial image regionsof the reference image with the relative positions being unchanged, therelative positions being the relative positions of the partial imageregions with respect to the standard point, to calculate the differencebetween the obtained state of the pixel values of the partial imageregions and the state of the pixel values of the partial image regionsof the standard image.